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The general purpose of an evaluation workshop is to 



orient school and state department of education personnel to the 
basic principles, procedures, and problems associated with evaluating 
educational programs and to the kinds of information an evaluation 
can provide for educational decision making. The workshop studied is 
based on the general evaluation model developed at the Center for the 
Study of Evaluation at UCLA. It takes two days to run and is ideally 
suited for groups of 30-45 participants. Workshop materials consist 
of an exportable leader's manual, a notebook for each participant, 
and a set of exercises for each team of three participants. Each 
module of the workshop involves instruction in a facet of evaluation, 
practice in solving relevant problems in this area, and feedback and 
discussion of the correct answers. By the end of the workshop, 
participants have completed exercises involving the selection, 
collection, analysis, and reporting of evaluation information for 
decision making. The development of the workshop began in January, 

1969, as an "opportunity project" within the Center. Three 
feasibility studies of it were completed between March, 1969, and 
April, 1970. A revised version was developed and field tested at five 
sites with the target audiences during the Spring and early summer of 

1970. Although the results of these and the previous sessions were 
positive, it was evident that changes were needed. These changes were 
made, and the workshop was given two special field trials at the end 
of August. After additional revisions were made, the workshop was 
given operational field tests throughout the United states between 
October, 1970 and August, 1971, with the target populations. The 
results of these field trials and a subsequent follow-up impact study 
were positive. (Author/CK) 
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This report describes the development and field testing of Evaluation 
Workshop I; An Orientation , a product of the Center for the Study of Eval- 
uation (Marvin C. Alkin, Director). The writing and rewriting of this 
workshop was supported by Center funds. Almost all the field-testing ac- 
tivities, however, were supported by the school districts, state departments 
of education, regional laboratories, and centers that requested the work- 
shop. Their assistance in this activity along with the aid of many non- 
program staff at the Center who conducted workshops was, of course, most 
appreciated and also ensured the realism of the field-test conditions. 

The Center staff primarily responsible for writing and editing the 
workshop were Dr. Stephen Klein (Program Director), Mr. David Churchman, 
and Mr. James Burry. Professor Marc -Andre Nadeau performed the statistical 
analyses of the field-test data and Dr. F. K. Heussenstamm was responsible 
for conducting the impact study. Dr. Eugene Grigsby assisted in writing 
and presenting the initial version. Dr. Fred Niedermeyer of the Southwest 
Regional Educational Laboratory provided valuable consultant help in the 
construction of the workshop materials, tests, and questionnaires. 

Dr. Rex Hagans of the Northwest Regional Educational Laboratory also con- 
tributed to the workshop's development by coordinating and conducting 
many of its field tests. Finally, a special note of thanks is due to the 
600 people who participated in the 22 field tests and who provided the • 
criticisms and data needed to improve the workshop and validate its effec- 
tiveness. 
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ABSTRACT 



ERIC 



The general purpose of Evaluation Workshop I: An Orientation is to 

orient school and state department of education personnel to the basic 
principles, procedures, and problems associated with evaluating educa- 
tional programs; and to the kinds of information an evaluation can pro- 
vide for educational decision making. The workshop is based on the gen- 
eral evaluation model developed at the Center for the Study of Evaluation 
at UCLA. It takes two days to run and is ideally suited for groups of 
30-45 participants. The workshop materials consist of an exportable 
leader’s manual, a notebook for each participant, and a set of exercises 
for each team of three participants. Each module of the workshop involves 
instruction in some facet of evaluation, practice in solving relevant prob- 
lems in this area, and feedback and discussion of the correct answers. By 
the end of the workshop, participants have completed exercises involving 
the selection, collection, analysis, and reporting of evaluation infor- 
mation for decision making. 

The development of the workshop began in January, 1969, as an "oppor- 
tunity project" within the Center. Three feasibility studies of it were 
completed between March, 1969, and April, 1970, although there was a nine- 
month period in which no work was done on its development due to other 
commitments of the project's staff. A revised version was developed and 
field tested at five sites with the target audiences during the Spring 
and early Simmer of 1970. Although the results of these and the previous 
three sessions were quite positive, it was evident that a number of major 
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changes were still needed before further field testing. These changes were 
made and the workshop was given two special field trials at the end of August, 
one involving professional staff from the Center, and the other personnel from 
the U.S. Office of Education and the National Science Foundation. After the 
necessary revisions were made, the workshop was then given 12 operational 
field tests throughout the United States between October, 1970 and August, 
1971, with the target populations. The results of these field trials and a 
subsequent follow-up impact study were quite positive. 

During the operational field testing, bids were requested for the publi- 
cation and dissemination of the workshop. CTB/McGraw-Hill of Monterey, Cal- 
ifornia, was subsequently chosen as the winning bidder and they began publi- 
cation of the final version in August, 1971. 



DESCRIPTION 



Evaluation Workshop I: An Orientation was developed at the Center for 
the Study of Evaluation at UCLA under the direction of Dr. Stephen P. Klein. 
The workshop is designed to orient school, project, and state department of 
education personnel to the basic principles, procedures, and problems asso- 
ciated with evaluating educational programs; and to the kinds of information 
an evaluation can provide for educational decision making. Thus, it is in- 
tended to be a comprehensive survey of the major facets of evaluation rather 
than an in-depth training program in evaluation procedures. (The Center's 
Training Materials Program is currently developing the latter kind of in- 
structional packages for which Workshop I would serve as an introduction.) 
Further, the language used in the workshop is appropriate for an audience 
with little or no background in program evaluation. 

The specific goals of the workshop are to train the participants to: 

1. Recognize a proper sequencing of major evaluation activities. 

2. Know when data selection, collection, analysis, and reporting 
procedures and evaluation techniques are. properly and appro- 
priately used. 

3. Know the kinds of information that should be generated by each 
type of evaluation activity and recognize whether or not reports 
contain such information. 

4. Recognize where and when evaluation information is needed for 
different kinds of educational decision making. 

5. Understand the different responsibilities and roles of the 
evaluator and project director. 

6. Understand the kinds of problems that may result from inade- 
quate program planning as they relate to evaluation activities. 
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To achieve these goals, the participants are grouped into three-member 
teams. Each team plays the role of the evaluator in the simulated evaluation 
of a 10th grade biology-ecology course. Since the workshop is organized in 
terms of the Center's evaluation model (Alkin, 1969; Klein, Fenstermacher , 6 
Alkin, 1971) , the participants are involved in conducting the needs assess- 
ment, planning the program and its evaluation, determining whether the pro- 
gram was implemented properly, assessing whether it is making progress to- 
wards meeting its objectives and what might be done to improve it, and, fi- 
nally, assessing and reporting its effectiveness. 

The basic instructional procedure for each of the workshop's modules 
involves receiving instruction in one of the five phases of the Center's eval- 
uation model (such as needs assessment) , practice in solving relevant problems 
in this area via a team exercise (such as determining the relative priorities 
among potential program objectives) , and feedback and discussion of the cor- 
rect answers. Instruction is provided via pamphlets, lectures, and audio 
tapes of conversations. Further, by the end of the workshop, participants 
have completed exercises involving the selection, collection, analysis, and 
reporting of evaluation information for decision making. 

The workshop leader's materials consist of an audio tape and a detailed 
and exportable manual. The only required training for a leader is previous 
participation in a workshop; however, the more evaluation experience he has, 
the better he will be able to answer the participant's questions. The team's 
exercises and feedback materials are packaged in a rip-off pad so that each 
team has a copy of the exercises and each participant has a copy of the in- 
structions and feedback materials. Each participant is also given a three - 
ring looseleaf notebook containing all the instructional and simulation mate- 
rial. Since the exercise instructions and feedback materials are also three- 
holed punched, the participant can put them in his notebook at the end of 
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each exercise. Thus, at the conclusion of the workshop, each participant takes 
home a 100 page guide to general evaluation principles and procedures that can 
serve as a valuable reference tool. 

The workshop takes two days to conduct and is ideally suited for groups of 
30-45 participants, although it may easily be run for larger groups. Exclusive 
rights for the publication and distribution of the workshop have been assigned 
by the Regents of the University of California to CTB/McGraw-Hill , Del Monte 
Research Park, Monterey, California 93940. 
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DEVELOPMENTAL HISTORY 

Initial Development 

In December of 1968, the California Educational Research Association 
(CERA) requested that the Center conduct a one and one -half day presession 
on evaluation prior to its March, 1969, Convention in Los Angeles. The Center 
accepted this invitation primarily because it would provide a forum for ex- 
amining its initial attempts at constructing a general evaluation model. 
Further, it was felt that this activity would not take a major portion of 
staff time away from other Center projects since it was believed that an 
existing training package, the Simulated Evaluation Exercise (Klein, Church- 
man, § Alkin, 1969), could be modified for the CERA presession. On further 
inspection, however, it became evident that the existing package was too 
limited in scope relative to the Center’s emerging evaluation model and much 
too time consuming for the needs of the presession. It was decided, there- 
fore, that the Center would devote a small portion of its discretionary 
funds for the development of a new workshop package. This, in turn, led to 
the construction of the first version of Evaluation Workshop I; An Orien- 
tation . 

Feasibility Testing - Version #1 

The first feasibility study of Version #1 of the workshop was conducted 
for fifteen participants at the CERA presession in March, 1969. Although 
only subjective participant questionnaire data was obtained regarding the 
workshop’s effectiveness, it was evident that the participants felt that the 
workshop was worthwhile . It was also evident that a number of changes in 
content, format, and administrative procedures were needed, such as the de- 
letion of presentations of information via lengthy lectures. Appendix A 
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contains a discussion of these changes and the results of the first 
feasibility test. Staff commitments to other projects, however, delayed 
further development of the workshop until 1970. 

During January and February of 1970, many of the changes noted in 
Appendix A were completed. This activity was stimulated by an invitation 
from CERA to again hold a one and one-half day presession for their con- 
vention. Thus, the second feasibility study was conducted in March, 1970, 
in San Francisco for fifteen participants. Questionnaire results were 
again quite positive, but it was evident that major changes were still 
needed. These changes focused upon reducing technical jargon, adjusting 
the evaluation model (and thus the sequencing of materials) to make it more 
field relevant, including a post-workshop questionnaire and two forms of 
a test for pre- and posttesting, developing an exportable leader’s manual, 
and related administrative details (e.g. , it was apparent that all team 
exercises should be printed on a different color paper than the instruc- 
tional materials given to each participant) . 

The changes noted above were made in the latter part of March and the 
third feasibility test was conducted on April 1, 1970. This one day work- 
shop was sponsored by the San Diego PACE Center and involved 75 school 
administrators. By almost any standard, one must conclude that this fea- 
sibility test did not go well. The acoustics of the room and the quality 
of the tape recordings were poor, the hastily revised exercises had many 
procedural flaws and misprints, there was not enough time to do all the 
exercises or read the instructional materials for them, and most of the 
participants came with the expectation that they would hear a series of 
lectures rather than be asked to roll up their sleeves and work. Having 
75 participants at the beginning of the workshop (one- third left after 
lunch) further compounded these problems. In short, it was a fiasco as 
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may be seen from the test and questionnaire results presented in Appendix B. 

On the positive side, however, it was clear that most of the procedural and 
content problems could be overcome relatively easily. Two members of the North- 
west Regional Educational Laboratory (who observed the San Francisco CERA 
presession and participated in the San Diego workshop) shared this optimism 
regarding the workshop's potential. Thus, at the Center's invitation, they 
agreed to help coordinate and conduct further field tests in the states 
included in the Northwest Region. 

Field Testing - Version #2 

As noted above, the San Diego feasibility study identified numerous 
problems in the materials. The necessary changes were made during the spring of 
1970 and ranged from the weight and color of the paper used for the exercises 
to major modifications in the sequencing and content of the workshop's instruc- 
tional modules. Sample answers also were included in the exercises to help 
the participants understand what was expected of them. Further, a new question- 
naire and three new forms of a test covering the workshop were developed for 
evaluation purposes. A typical item from one of these tests appears below: 



Evaluator A suggests assigning children randomly to the new experimental 
5th grade mathematics program and to the old program. He wants a posttest 
given at the end of the year to determine how much learning took place . 

Evaluator B suggests giving the new experimental program to one existing 
class and the old program to another existing class. He wants a pretest 
given at the beginning of the year and a posttest given at the end of the year. 

On your answer sheet, indicate your opinion as to the relative merits of the 
two designs in providing the most information about the quality of the new 
program. 

a. Evaluator A's design is better than B's. 

b. Evaluator B's design is better than A's. 

c. Either A or B; both are very good. 

d. Neither A or B; both are very poor. 
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A one -day preview of the workshop was held in Portland, Oregon for the 
24 people involved in coordinating the activities of NWREL and each of the 
areas it serves. The purpose of this session was to enlist their aid in 
obtaining field-test sites for the workshop. As a direct result of this 
session and the efforts of the NWREL, field tests were eventually conducted 
in each area. 

During June and July of 1970, a total of five field tests of Version #2 
were conducted by project staff, representatives of the NWREL, and private 
agencies. Hie participants in these sessions were school administrators, 
project directors, and curriculum supervisors. The results of these field 
tests were quite positive (see Tables 1 and 2 , and Appendices C and F) and 
instrumental in obtaining a favorable bid for the workshop’s publication. 

These field tests did indicate, however, that a number of important changes 
were still needed before the workshop could be considered truly exportable. 
These changes were made during July and August of 1970 and included: a 
complete revision of the program planning module to reduce reading time and 
increase participation by having each team member read only one of three 
instructional booklets prior to a team exercise that involved using the 
information contained in all three booklets; a total modification of the 
leader and participant materials so that three-ring loose-leaf notebooks 
could be used to store all the participant’s instructional and simulation 
infoimation as well as copies of the feedback materials; an expansion of the 
context of the simulation so that it would have a broader appeal, including 
a module on reporting evaluation results to lay audiences; and deciding that 
the workshop needed at least two days to be conducted properly. The change to 
three -ring binders also essentially eliminated the need for team coordinators, 
i.e., one leader could now conduct the workshop for over 100 people. 

Once these changes were completed, two special field tests were 
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conducted. These field test differed from the previous ones with school 
administrators in that they involved individuals with expertise in evaluation. 
The first of these special field tests was conducted at the Center for its 
professional staff and the second was held in Washington, D.C. for represen- 
tatives from the United States Office of Education and the National Science 
Foundation. As a result of these sessions, some of the exercises and 
instructional modules were modified to ensure their utility in providing 
appropriate instruction and realistic practice in solving a broad range of 
evaluation problems. These changes constituted the last major revisions of 
the workshop prior to its publication by CTB/McGraw-Hill . 

Operational Testing - Version #3 

Workshop I was given 12 more field tests between September, 1970, and 
August, 1971. These sessions were held for the target audiences of school 
and state department of education personnel throughout the United States. 

The results of these field tests were again quite positive, especially consider- 
ing the fact that most of them were not conducted by project staff or even 
by trained evaluators (see Tables 1 and 2 and Appendices E and G) . Concur- 
rent with these field tests, the Center accepted the bid from CTB/McGraw-Hill 
of Monterey, California, for the publication and diffusion of the workshop. 
Center staff then worked with CTB/McGraw-Hill in making the final revisions 
in the materials based on the operational field test results. 

During June of 1971, the Center conducted an impact study involving all 
the people who participated in field tests of Versions #2 and #3 prior to 
May, 1971. The results of this impact study appear in Table 3 and in 
Appendices H, I, and J. 

Publication of Evaluation Workshop I: An Orientation in August of 1971 
by CTB/McGraw-Hill brought to a close the Center's formal responsibility for 
the workshop's development. 
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TEST RESULTS 

Version #1 

Two forms (A and B) of a test covering the workshop’s objectives were 
administered as part of the San Deigo feasibility study. Although a small 
but statistically significant increase in performance was obtained, the results 
were not especially encouraging. Appendix B contains a more detailed 
discussion of the results and an analysis of the test data. 

Version #2 

Three forms of a 23 item test were constructed to cover the material in 
Version #2. In order to maintain test security and counterbalance forms within 
the constraints of administrative feasibility, the following procedures were 
used: on pretesting, participant A took form A, participant B took form B, and 

participant C took form C; on posttesting, participant A took form B, B took 
form C, and C took form A. This meant that at each team’s table, three different 
forms were used for both the pretest and the posttest; and each participant had 
different forms for pretesting and posttesting. A typical item from these 
tests appears below: 

Which of the following two activities 
should be done first? 

a. instituting a new program 

b. deciding how the program will be 
evaluated 

The summary test results for Version #2 are presented in Table 1 and more 
detailed analyses for each field test site are presented in Appendix C. An 
inspection of these data indicates that there was a very definite increase in 
performance between pretesting and posttesting at the .01 level. The results of 
the two special field trials with personnel from the Center, U.S. Office of 
Education, and the National Science Foundation are presented in Appendix D. 
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TABLE 1 



Summary of Test Results for Versions #2 and #3. 





Version #2 




Version #3 




(5 sites) 




(12 sites) 


Mean 

Pretest 

Score 


14.70 




14.69 


Mean 

Posttest 

Score 


16.80 




17.15 


Mean 








Change 

Score 


+2.10 




+2.46 


"t" test 
Significance 


4.85 




12.67 


Level 


.01 




.001 


Number and Percent 
of participants who: 








Increased 


58 (761) 


244 


(794) 


Did not change 


5 ( 71) 


21 


( 74) 


Decreased 


14 (171) 


44 


(144) 


Total: 


77 


309 
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Version #3 

The same set of tests and testing procedures were used with Version #3 as 
were used with Version #2. The major reasons for this were time constraints 
in getting a more relevant set of measures constructed and the desirability of 
using a consistent set of assessment instruments across field tests. Since 
there were many changes between the two versions in both administrative 
procedures and content, the three test forms constructed for Version #2 
were not as congruent with objectives defined for Version #3 as they should 
have been. Thus, any differences in performance between pretesting and post- 
testing in the Version #3 studies are conservative estimates of the true 
increases in performance relative to differences obtained in the Version 
#2 studies. Despite this conservative bias, the test results with Version 
#3 were more positive than they were with Version #2. Table 1 contains a 
summary of these results and Appendix E contains an analysis of the pretest- 
posttest results for each of the 12 field test sites. 

QUESTIONNAIRE RESULTS 

At the conclusion of each workshop, a questionnaire was administered 
to the participants so that they could both evaluate the workshop and make 
suggestions for its improvement. The results of these questionnaire studies 
were generally quite favorable. 

Version #1 

Despite the many problems with the various editions of Version #1, 
the participants generally felt that it was a worthwhile experience. For 
example, when the 15 participants at the very first workshop were asked the 
question "Did the workshop improve your understanding of the evaluation 
process none of them said "none or veiy little," three said "some," and 
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12 said "a great deal." Appendices A and B contain more detailed analyses of 
these early questionnaire results. 

Versions #2 and #3 

Although slightly different questionnaires were used with the two versions, 
they were sufficiently alike to make comparisons between them. Table 2 
contains the summary results for the two versions for five of the key items in 
the post-workshop questionnaires. An inspection of this table indicated that 
both versions of the workshop were considered quite valuable by most of the 
participants. For example, 88% of the participants indicated that they 
developed solutions to their evaluation problems at the workshop , 87% said 
that the overall quality was at least "good," and 94% said that they would 
recommend it to others. 

A comparison of the results between Versions #2 and #3 indicated that the 
latter had a somewhat more favorable impression on the participants. For 
example, 97% of those receiving Version #3, compared to 62% receiving Version 
#2, said that they found at least some solutions to their evaluation problems. 

It appeared, therefore, that the changes in format and content between the two 
versions did have a positive effect upon the participants’ evaluation of the 
workshop . 

Appendices D, F, and G contain more detailed analyses of the complete 
questionnaire results from each field test site. Appendix K contains the 
results of a follow-up questionnaire that was developed, administered, and 
scored by Dr. M. E. Hickey of the Seattle Public Schools after NWREL ran 
Version #2 of the workshop in his district. 

IMPACT STUDY 

In Spring of 1971, a study was initiated to determine the impact of 
Workshop I on the participants who had taken Version #2 or #3. In order to 



O obtain a sufficient interval between the workshop and the follow-up, the study 

FRir 




was limited to workshop sessions held prior to April, 1971. However, since the 
test and questionnaire data from the excluded sites (Newport II and III, 

Texas, and Hawaii) were even more positive than those obtained with the sites 
included in the impact study, it may be assumed that the limitations noted 
above would only result in a conservative bias. 

A total of 297 questionnaires were mailed out on May 15, 1971. By 
June 9, 253, or better than 851 were returned. This high percentage was 
achieved as a result of sending out reminders, making telephone calls, and 
related follow-up procedures. 

Table 3 contains a summary of the results of the impact study and 
Appendices H, I, and J contain detailed analyses by version and type of 
participant. These data indicated that both versions of the workshop were 
considered valuable by most of the participants for whom it was designed. 

For example , 901 of all the participants who took it indicated that it 
enhanced their understanding of evaluation problems. Similarly, of the 141 
participants who felt the workshop was relevant to their job responsibilities, 
90 of them indicated that it helped them use their time, money, or resources 
more effectively. The workshop was especially well received by participants 
with job responsibilities primarily in the areas of evaluation and research; 
80% of the individuals in this group said that they used their notebook after 
the workshop. 

An inspection of Table 3 also reveals that Version #3 had a somewhat 
greater impact than Version #2 even though the time between the session and 
the follow-up was less. For example, 78% of the participants who had Version 
#2 and 93% of the participants who had Version #3 said that the workshop 
enhanced their understanding of evaluation. 

One indirect but important indication of the workshop's impact was the 
number of times it was requested to be run for a group by someone who had 



observed or participated in a previous workshop. For example , Dr. Mike Hickey 
of the Seattle Public Schools had it run twice in his district, Dr. James Freda 
of the Orange County PACE center had it run twice (Santa Ana and Cypress), Mr. 
Robert Otto of the Newport-Mesa Unified School District sponsored three work- 
shops after his initial contact with the materials during the San Diego work- 
shop, and all the coordinators who attended the preview session at the NWREL 

i 

arranged for workshops in their areas. A related indication of this type of 
impact has been the numerous requests by sponsors of Workshop I to field test 
new workshops and training materials produced by the Center. 



TABLE 3 Continued 
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CERA PRESESSION FIELD TEST REPORT 



LOS ANGELES: CENTER FOR THE STUDY OF EVALUATION 



DATE: MARCH 18, 1969 



TO 



FOR THE RECORD 



CC: M. ALKIN 

M. BENTZEN 

R. SKAGER / 

FROM: S. KLEIN 

G. GRIGSBY 

SUBJECT: COMMENTS ON CERA PRESESSION AND SUGGESTIONS FOR 
FURTHER DEVELOPMENT OF THE SIMULATED EXERCISE. 



A. Summary 

The presession went well. Almost all of the participants 
thought it was very worthwhile. The cycle of non-threatening 
work and then feedback appealed to the participants and it 
helped them identify with the evaluation problem, i.e., they 
saw it as solvable and they participated in figuring out ways 
to solve it. Two factors limited the success of the game, 
namely: (1) the rush to get materials completed and (2) the 
mobility of the participants in and out of the sessions. 
Suggestions for further development of the game also are discussed 
below. Most of these are designed to make the game more trans- 
portable via improving the form of the feedback materials . 

B. Evaluation of the presession. 

1. At the end of the training session, an evaluation sheet was 
^ completed by all participants (see attached table) . An in- 

spection of their evaluations and comments indicated that 
they thought the session was well run, very worthwhile, and 
pitched to their level of understanding. 

2 . Problems with running the presession 



a. Time to prepare it was limited. Thus, sane of the 
materials must be revised and others written out (such 
as Chase’s talk and the Skager-Kleih "debate"). 

b. The set of the participants was to hear lectures. They 
were not prepared for work sessions in the sense that they 
made appointments during meeting times, arrived late, left 
early, etc. Despite this problem, the actual number of 
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participants increased over the two day period. We had a 
running total of 10-16 during the two days. 

c. The game is still not transportable in that we had to field 
several questions that were directly or tangentially relevant 
to the game. Most of these were asked during the "feedback" 
discussions. It also was important for us to periodically 
"sit in" on the group meetings to help and direct their dis- 
cussions . 

d. By means of the application forms, we assigned at least one 
"strong" design person to each group. This improved the 
groups' efficiency. 

e. The various units of the sessions took about the time we 
alloted for them; however, the mobility of the partici- 
pants frequently delayed getting started and caused us to 
end early on the first day. A few reviews were inserted 
informally to further cope with the mobility problem. 

C. Suggestions for immediate and long term development of the game. 

1 . , Immediate 

a. The various talks should be typed from the tapes and then 
edited and revised as necessary. 

b. A good final report should be written or at least one that 
points out the weaknesses in the present one, i.e. , written 
rather than oral feedback. 

2. Long term 

a. Prepare a shorter but more complete set of advance sum- 
mary materials on various approaches to evaluation. We 
also may wish to attach sane discussion questions to 
these. These should be short enough as to not preclude 
their being read, but long enough to convey the "theory's" 
major thrust. 

b. Insert the unit on objectives immediately after the for- 

mal request for the evaluation. This unit should be re- 
vised to include the following: an instructional package 

on writing objectives, sample objectives and items to be 
critiqued, and written feedback in the form of a legiti- 
mate set of objectives and items with an explanation of 
how they differ from the old ones. The revised set can 
then be used by the participants and would make the game 
more realistic. The achievement and attitude items 
might be split for this unit so as to show them how 
easily it is to forget to include attitudes. 
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c. Clean up all feedback to make sure that it meets the 
following criteria: 

(1) It is written. 

(2) When relevant (e.g. , revised letter of understand- 
ing) , it contains an explanation of how it differs 
from the draft material. 

(3) It simulates versions of the real thing. These may 
be condensed versions, e.g., the final report. 

(4) The feedback materials become a part of the game, 
e.g., the objectives of the biology curriculum. 

(5) Whenever possible, it contains various appropriate 
and inappropriate alternatives to solving the prob- 
lem, but with a clear indication as to which of 
these alternatives will be used in the remainder of 
the game. Remember, we want to encourage flexibility. 

d. Greater stress should be placed on the groups to reach a 
more finished product in each of the working sessions. 

For example, they might have to turn in their work. 

e. The groups' work could be evaluated in tenns of how well 
they did in each work session. Such information would 
be invaluable for improving the game, but it was not 
possible to collect in our presession due to the mobility 
and limited time factors. 
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TABLE 1 

Summary of Evaluations 



Item 



. Improve your ability to 
conduct an evaluation. 

a. None or very little 

b. Some 

c. A great deal 



Improve your ability to 
judge the quality of an 
evaluation report and/or 
procedure. 

a. None or very little 

b. Some 

c. A great deal 



Improve your understanding 
of the evaluation process. 

a. None or very little 

b. Some 

c. A great deal 



Alert you to special evaluation 
problems and ways of handling 
them. 

a. None or very little 

b. Some 

c. A great deal 



Number 



1 

4 

9 



1 

3 

11 

0 

3 

12 



1 

6 

8 



&**!<* yr^r- 



r 



» ■ 'J* 



Item 


Number 


i 

In terms of your own abilities, 
the presession was 




a. Too naive 


1 


b. About right 


13 


c. Too sophisticated 


1 


Should the presession in its 
present foimat be held again? 




a. No 


0 


b. Yes, but with extensive 
changes 


1 


c. Yes, but with few changes 


4 


d. Yes 


10 
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Question 



1. Improve your ability to 
conduct an evaluation 



2. Improve your ability to 
judge the quality of an 
evaluation report and/or 
procedure . 



3. Improve your understanding 
of the evaluation process. 



4. Alert you to special evaluation 
problems and ways of handling 
them. 



5. In terms of your own abilities, 
the presession was. . . 



Comments 



"Elements of instruction down-to- 
earth, concrete, generalizable." 

"Completely changed from old 'school 
survey- type' to system oriented, etc." 

"Broadened my understanding." 



"Popham's paper excellent." 
"Developed my critical judgment." 



"A great deal. . .discussion of Skager 
of program design and what could be 
called - evaluation continuum." 

"I knew much of the specifics - 
steps in evaluation, etc., - from 
prior reading." 

"Clarification of evaluator's 
role and procedures. Skager 's 
presentation and first day's 
presentation enlarged my back- 
ground and helped me identify 
and clarify my own techniques." 



"Tape describing former evalua- 
tion extremely well done." 

"Taught well by Dick Watkins." 

"I had no idea of some of the 
problems suggested and ways of 
handling them." 



"About right. Excellent - met 
individual needs too - talked to 
each in terms of his questions 
and level of sophistication'.' 

"About right - in terms of pace, 
model used. Too sophisticated in 
terms of statistical background 
expectations of participants." 

"Excellent presession. Good ma- 
terials - continuity - and well 
presented." 



Question 



Comments 



5. (continued) 



6. Should the presession in its 
present format be held again? 



✓ 



7. What changes would you recommend 
be made in the presession if it 
were to be given again? 



"About right. I would not have 
wanted it simple nor more sophisti- 
cated." 

"Organization and presentation 
highly effective- -and very practi- 
cal in orientation- -without talking 
down to anyone- -first 2 day meeting 
with continuity I've ever partici- 
pated in." 

"I like the simulation method very 
much- - realistic- -practical - - covered 
a lot of ground." 

"I think planning the work groups 
more specifically in terms of per- 
sons would be helpful." 

"I would have liked more data on 
our ' target district ' to add more 
flexibility to our choice options." 

"Same format." 

"Yes. Far superior to the usual 
lecture format! The simulated pro- 
ject and the techniques used (inclu- 
ding the role playing and the tape 
as examples) were outstanding. This 
alone was worth the time although it 
was a concommitant gain." 

"Yes, but with a few changes. 

More information on type of 
session before commitment is made by 
prospective participant. This sec- 
tion was excellent." 

"1. Be sure material which is mailed 
to some participants is mailed to 
all beforehand. 

2. Summarization of material presented 
by resource personnel." 

"None at this time. The whole program 
was great!!" 




Question 



Comments 



7. (continued) ' ! None--one of the best, if not the 

best that I have attended. Too many 
presessions fractionate their material 
into different areas that, while 
important, subtract from the over all 
process under consideration. Very 
worthwhile- -since many of us get tied 
up in so much detail we lose or fail 
to consider many of the important 
points brought out. Thanhs again." 

"It is difficult to anticipate- -and 
you tried to do it- -careful formation 
of discussion groups. I think this 
session might have spent an hour or 
so on a discussion of measures --al- 
though it would be difficult to limit 
the discussion. This presession 
really benefited from imposed res- 
trictions." 

"I was very satisfied. Would like 
a roster rundown on name, address, 
job, phone, of all members of group. 

A fringe benefit." 

"Beautifully organized. Telescoped 
a great deal of learning into a 
brief period. Similar to programmed 
learning, but much much better!" 

"The presession was excellent! It 
answered my needs --and expectations. 

I appreciated the relaxed atmosphere 
created by the 'easy' style of the pre- 
senter- -also the variety of techniques 
used. Very planned and organized!" 

"I feel this presession was very well 
presented and would really not need 
any changes. Thank you." 
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LOS ANGELES: CENTER FOR THE STUDY OF EVALUATION 



DATE: 


June 22, 1970 


TO: 


S. Klein 


FROM: 


M. Nadeau 


CC: 


D. Woolley 




J. Burry 



SUBJECT:. Analysis of Pre- and Posttest Results - San Diego 
Evaluation Workshop - 4/1/70 
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UNIVERSITY OF CALIFORNIA — (Letterhead for interdepartmental u«a) 



ANALYSIS OF PRE- AND POSTTEST RESULTS 



San Diego Evaluation Workshop - 4/1/70 

Results 

Table 1 presents the performance of the group of participants who 
took both the pre - and the posttests without considering the form they 
took (A or B) . 

Table 1; Performance of the Group: Pre - and Posttests 



Frequency Distribution 




Mean 


Median S.D. > 

i 

j 




Pre 


Post 


Pre 


17.15 


17.5 2.37 \ 

j 


22-23 


0 


8 






] 

i 

) 


20-31 


9 


8 


Post 


17.95 


18.1 3.14 j 


18-19 


12 


10 


"t" (.01) 


= 8.69 


I 

(significant) ] 

t 


16-17 


13 


10 


r = .0574 




j 

j 


14-15 


9 


6 








12-13 


3 


3 








10-11 


0 

46 


1 

15 “ 









If we look simply at the mean gain showed by the group (Table 1) from 
the pre - to the posttest (.80), we might be led to believe that the train- 
ing has little or no effect on the participants. Even though the "t” value 
is highly significant (8.69, p<.01) statistically, intuitively this value 
has little meaning. 

Not only does the training seem to have little effect, but in addi- 
tion, rather than increasing the homogeneity of the group, the expected 
result, the training seems instead to increase the heterogeneity of the 
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group as shown by the S.D. (Pre: 2.37 and Post: 3.14). There is a very 

little relationship between the pre- and the posttests as shown by the low 
correlation coefficient, r = .0574. 

However, limiting the interpretation of the results simply to the 
means and S.D. would lead us to arrive at false conclusions. A more 
encouraging outcome is apparent when we consider the performance of the 
individuals on both the pre- and the posttests. (See Table 2) In fact, 
the results show that 25 out of 46 participants (541) show an increase, 

16 out of 46 (35%) show a decrease, and 5 out of 46 (11%) achieved the 
same score on both tests. 

Table 2: Performance of the Group: % of Increase 

Increase: 25 out of 46 54% 

Decrease: 16 out of 46 35% 

Same : 5 out of 46 11% 

The percentage of increase (54%) for the group as a whole does 
represent a positive increase considering the fact that the workshop 
was conducted in an extremely limited period of time. It would have 
been more encouraging to have a higher percent of gain, but considering 
the time constraints, the gain is positive. Among other possible ele- 
ments that could explain the small gain achieved by the group, we could 
suggest the element of fatigue. 

We might, howevever, be concerned about the fact that 35% of the 
participants show a regression on their score from the pre- to the post- 
tests. But in fact, we do not know for sure what this outcome means; 
does it represent a real detrimental effect of the training session, an 
effect we seriously doubt, or does it represent a testing effect? The 
latter element appears to us to be the key to the problem. 
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Testing Effect 

Thirty -five percent of the group as a whole shows a regression 
from the pre- to the posttests. If we look at the individual performances 
(Table 3) , we see that 13 out of 16 people who regressed took form B as 
pretest and form A as posttest, and that only 3 out of 16 participants who 
regressed took form A as pretest and form B as post test. 

Table 3: Performance of the Two Subgroups: % of Increase 



A then B 


Increase: 


20 out of 25 


80% 




Decrease: 


3 out of 25 


12% 




Same : 


2 out of 25 


8% 


B then A 


Increase : 


5 out of 21 


24% 




Decrease : 


13 out of 21 


62% 




Same : 


3 out of 21 


14% 



The above results, when the group is divided into two subgroups 
(one being composed of those who took form A then form B and the other 
being composed of those who took form B then form A) , lead us to believe 
that form A is the more difficult one. If we look at the results in 
Table 4, we see that the "A then B" group shows a relatively large increase 
(2.40) and the M B then A M group shows a decrease (1.09). 
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Table 4: Performance of the Groups ; 

A then B , B then A : Pre- and Posttests 

B then A group 
Pre 



Frequency Distribution: 
A then B group 





Pre 


Post 


22-23 


0 


5 


20-21 


2 


8 


18-19 


6 


4 


16-17 


7 


3 


14-15 


7 


3 


12-13 


3 


2 




25 


25 




Pre 


Post 


Mean 


16.28 


18.68 


S.D. 


2.3 


2.97 



22-23 


0 


20-21 


7 


18-19 


6 


16-17 


6 


14-15 


2 


12-13 


0 


10-11 


0 




21 




Pre 


Mean 


18.19 


S.D. 


2.02 



Post 

3 

0 

6 

7 

3 

1 

1 

21 

Post 

17.10 

3.17 



The results are much more impressive when we consider the individual 
performances. (See Table 3) Among those who took test A then B, 20 out 
of 25 show an increase (80%) , 3 out of 25 show a decrease (12%) , and 2 
out of 25 achieved the same score (8%) . Among those who took test B 
then A, 5 out of 21 (24%) show an increase, 13 out of 21 (62%) show a 
decrease, and 3 out of 21 (14%) achieved the same score. 

Thus, it appears that giving form B as pretest and form A as post- 
test was a disadvantage for the people of this group, since form A is much 
more difficult than form B. ' * ’••••■ 
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Conclusion 

We can conclude that the training session on evaluation held in 
San Diego was effective even though the results do not reflect a very large 
increase. The effect did. not appear because of the level of difficulty 
of one of the two instruments used to measure this effect. 

We suggest, therefore, that if both of these forms are to be used 
as pre- and/or posttest they should be revised to make them equivalent. 
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Appendix A: Indiyidual results on the pre- and posttests without 

considering the form given 



Pre 


Post 


Pre 


Post 




18 


10 -8 


17 


16 


-1 


20 


14 -6 


14 


20 




14 


16 


13 


20 




12 


14 


18 


16 


-2 


14 


15 


16 


22 




21 


22 


19 


18 


-1 


18 


22 


14 


18 




17 


17 


19 


18 


-1 


20 


22 


17 


12 


-5 


18 


21 


20 


14 


-6 


15 


18 


21 


17 


-4 


17 


13 -4 


17 


23 




14 


19 


21 


19 


-2 


13 


19 


20 


16 


-4 


18 


21 


15 


15 




16 


20 


15 


20 




20 


17 -3 


17 


13 


-4 


18 


18 


18 


22 




17 


17 


17 


21 




19 


22 


16 


19 




19 


20 


16 


19 




15 


15 


21 


17 


-4 


17 


23 


18 


16 


-2 
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Appendix B: 


Individual results on the pre- 
considering the form given 


and the posttests 






A 

Pre 


B 

Post 


B 

Pre 


A 

Post 




14 


16 


18 


10 


-8 


12 


14 


20 


14 


-6 


14 


15 


20 


17 


-3 


21 


22 


18 


17 




18 


22 


17 


17 




17 


17 


18 


16 


-2 


20 


22 


16 


22 




18 


21 


19 


18 


-1 


15 


18 


14 


18 




17 


13 -4 


19 


18 


-1 


14 


19 


17 


12 


-5 


13 


19 


20 


14 


-6 


18 


21 


21 


17 


-4 


16 


20 


17 


23 




19 


22 


21 


19 


-2 


19 


20 


20 


16 


-4 


15 


15 


15 


15 




17 


16 -1 


16 


19 




14 


20 


. 21 


17 


-4 


13 


20 


17 


23 




15 


20 


18 


16 


-2 


17 


13 -4 








18 


22 








17 


21 








16 


19 
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Number returned questionnaire = 37 



PARTICIPANT FOLLOW-UP QUESTIONNAIRE 
Evaluation Workshop 



I . Complete the table below by placing checks in each column to indicate your feelings 
about various evaluation tasks. 



TASK 



As a result of my attendance at the Evaluation Workshop: 

I am able to attack this I see immediate practical 
task more effectively application of the tech- 

niques demonstrated in the 
Evaluation Workshop. 





YES 


DOESN'T 

APPLY 


ABOUT 

SAME 


NO 

ANS 


YES 


DOESN'T 

APPLY 


NO 


NO 

ANS 


a. 


Determining the role of 
a project director 
during evaluation. 


10 


8 


14 


5 


9 


8 


12 


8 


b. 


Determining the role 
of an evaluator during 
evaluation . 


17 


3 


12 


5 


15 


3 


12 


7 


c. 


Conducting a Needs 
Assessment. 


13 


3 


13 


8 


15_ 


4 


10 


8 


d. 


Stating an instructional 
problem. 


16 


2 


U 


5 


16 


4 


11 


6 


e. 


Contracting for evalua- 
tion services. 


10 


12 


10 


5 


8_ 


12 


10 


7 


f. 


Preparing a project 
proposal. 


Cl2 


_5 


14 


5> 


13 


6 


1 1 


7 


g. 


Evaluating research 
designs as good, fair, 
or poor. 


10 


3 


19 


5 


12 


2 


15 


7 


h. 


Determining if a program 
is being conducted as 
planned. 


18 


1 


14 


4 


17. 


1 


13 


6 


i. 


Improving a program as it 
is being conducted. 


13 


4 


15 


5 


15 


_5 


n 


6 


j. 


Evaluating a project's 
Final Report. 


18 


3 


15 


4 


16 


4 


n 


6 

- 



O 



4 ? 



II. Please check each of the following items: Feel free to write additional comments 

beside any of them. 

1. Did you develop solutions to your evaluation problems at the workshop? 

12 YES 23 NO NO ANS: 2 

2. The overall quality of instruction at the workshop was: 

2 EXCELLENT 8 GOOD _9%_AVERAGE _8^_FAIR 7 POOR NO ANS: 2 

3. If the same workshop was held again, would you recommend that others attend? 

15 YES 19 N O NO ANS: 3 

4 . How good were the meeting room facilities for the workshop? 

0 VERY GOOD 14 GOOD 15 POOR 6 VERY POOR NO ANS: 2 

5. Do you think you had the appropriate prerequisites or prior knowledge to 
make what you learned at this workshop of use to you? 

11 MORE THAN NECESSARY 14 RIGHT AMOUNT 7_N0T ENOUGH NO ANS: 5 

6. During the workshop did you wish to discuss evaluation problems that had 
arisen in your own work? 

15 YES 19 N O NO ANS: 3 

7. If answer to item 6 above was M YES, M was there an opportunity to pursue 
this interest? 

0 QUITE A LOT £_SQME ,11 NONE AT ALL NO ANS: 3 

8. Did the amount of reading required during the workshop seem acceptable? 

15 TOO MUCH 18 JUST RIGHT 0_TOO LITTLE NO ANS: 4 

9. Did the amount of writing and problem solving required during the work- 
shop seem acceptable? 

11 T OO MUCH 19 JUST RIGHT 3 TOO LITTLE NO ANS: 4 

10. In your own job will you use what you have learned in the immediate 
future? 

19 Y ES 10 NO 6_NOT SURE NO ANS: 2 

11. Would you like to learn more about evaluation as defined in the work- 
shop? 

24 YES 11 NO NO ANS: 2 
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III. 



o 

ERIC 



12. Was the time allowed for the workshop sufficient to learn the materials? 

18 YES 17 N O NO ANS: 2 

13. The language level of the materials and the instruction in the workshop 
was: 

1 VERY DIFFICULT 4 DIFFICULT 26 ABOUT RIGHT 4 TOO EASY NO ANS: 2 



14. The pace at which the workshop was conducted was: 

9% TOO FAST 15 ABOUT RIGHT 9h TOO SLOW NO ANS: 3 



15. The sequence of activities during the workshop was: 

4 VERY CONFUSING 14 CONFUSING 17 EASY TO UNDERSTAND NO ANS: 2 

16. The content of the workshop is appropriate for: (check one or more) 

8 UNIVERSITY TEACHERS 13 OTHER TEACHERS 26 SCHOOL ADMINISTRATORS 
2 Project Planners ‘ 

6 O THERS ( 1 too elementary) 

1 independent evaluators NO ANS: 4 

% 

17. Your feeling during the workshop can best be described as: 

6 VERY FRUSTRATED ^FRUSTRATED 14 NEUTRAL 9_EAGER NO ANS: 1 

18 . How would you describe the correspondence between what you expected to get 
out of the workshop and what you actually did get out of the workshop? 

8 ABOUT WHAT I EXPECTED 

4 MORE THAN I EXPECTED 

24 LESS THAN I EXPECTED. NO ANS: 1 

What would you like to see changed in the workshop and how would you chatnge it? 
What would you add or delete? (Use backside of this sheet if needed.) 

General Comments 

Excellent presentation 

Good content, idea, approach, very timely 

The workshop was generally poor 

There was nothing new discussed - poorly handled 

O.K. as was 
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Scope of Workshop 

Teach less more thoroughly, more depth, more detail 

Too elementary, wrong level - should be more fundamental for application 

Less structure - more flexibility during day 

More emphasis on program evaluation without "funding" emphasis 

Needs survey inadequate 

Psychologist's viewpoint of a school need rather than team approach 
More relevant to specific needs of participants 

Organization 

Smaller groups, divided by various factors: 

elementary/secondary personnel 
advanced/beginners 
specific needs 

Range of experience of participants too broad 
Discussion in large group needed 
Smaller Audience 

Workshop was right only for project directors 

Facilities and Audio-Visual 

Criticism of room, acoustics 
Criticism of tape quality 
Criticism of use of tape at all 

"didn't achieve the 'realism' intended" tapes and reading 
detracted from task 

Printed Materials 

Useful alone (after workshop) 

Too much paper shuffling with little reading time; what small part 
was worthwhile should have been condensed 

Length of Workshop 

More time or less materials 
Felt rushed 
2-5 days needed 
4 hours enough 

Sessions were too long (with break) 

More time for small discussions 

More time for small groups to work problems by trial and error -- 
followed by evaluation of leader as to how group did 



O 

ERJC 
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Evaluation Team 



UCLA notables were not present 

Instruction was too slow, too casual, considering nature of need for 
meaningful evaluation 

Staff should be more organized and less prone to lazy attitude toward 
their role and responsibility 
More commitment on part of team 
Seemed like practice for graduate students 

Miscellaneous Comments 

Background information should be available before workshop 
Should have been made more relevant to Bilingual Projects 
I have my own model; I was exposed to your type of model in 
1930 at the University of Wisconsin and it was 
discarded -- back to the drawing board 
More information on how to evaluate a project: 

1. design 

2 . roles 

3. how to recycle . 



APPENDIX C 



VERSION #2 TEST RESULTS FOR EACH SITE 



ERLC 
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VERSION #2 TEST RESULTS FOR EACH SITE 





Santa Ana 
(Calif.) 


Seattle I 
(Wash.) 


Plumas 

(Calif.) 


Juneau 

(Alaska) 


Portland I 
(Oregon) 


Total 


Mean 

Pretest 


15.5 


14.4 


13.8 


15,5 


15.0 


14.7 


Mean 

Posttest 


15.4 


16.9 


17.3 


16.4 


16.8 


16.8 


"t" test 
results 




2.77** 


3.92** 


1.15 


5.4** 


4.85** 


N increase 


4 


14 


12 


9 


18 


58(76%) 


N same 


1 


2 


1 


1 


0 


5(7%) 


N decrease 


3 


4 


0 


3 


4 


14(17%) 



* .05 

** .01 



Note: Santa Ana and Juneau were run in 1 to lh days. The pther workshops were run 

in 2 days. 



APPENDIX D 



FIELD TEST RESULTS INVOLVING CENTER, USOE, 
AND NATIONAL SCIENCE FOUNDATION PERSONNEL 



FIELD TEST RESULTS FOR SPECIAL FIELD TESTS 





R 5 D 
Center 


Washington D.C. 


Total 


Mean 


pretest 


17.31 


14.27 


16.07 


Mean 


posttest 


19.63 


17.81 


18.89 


"t" test 


results 


3.83** 


4.42** 


5.66** 


N increase 


13 


10 


23 (84%) 


N same 


2 


0 


2 ( 820 


N decrease 


1 


1 


2 ( 820 



** .01 
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Table I: Distribution of M yes" answers to questions 1: "I am able to attack 

this task more effectively," per training session and total . 
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Table II: Distribution of "yes" answers to question 2: M I see immediate 

practical application," per training session arid total' . 
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Table III: Distribution of "Frequently and/or occasionally" choices 

question 3: "Within the next year I forsee referring' to 

and using this resource, M per training session and total. 




Table IV : Distribution of answers to Part III of the questionnaire , 

per training session arid total . 
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Table IV (continued) 
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14. Would you like to learn more Yes 
about evaluation as defined in No 
the workshop? 



Table IV (continued) 
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APPENDIX E 



VERSION #3 TEST RESULTS FOR EACH SITE 
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APPENDIX F 

VERSION #2 QUESTIONNAIRE RESULTS FOR EACH SITE 
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Table I : Distribution of "yes" answers to question 1; "I am able to attack 

this task more effectively,” per training session arid total. 
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APPENDIX G 



VERSION #3 QUESTIONNAIRE RESULTS FOR EACH SITE 
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Table II: Distribution of "yes" answers to question 

"I see iiimediate practical application 
per training session arid total . 
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Table IV (continued) 



1^01 


H vO vO to 

N H rt 
H r-1 


141 


100 


eg 


UOJJ 0 AB 9 H 


O O 00 00 
rH 


Cl 

rH 




to 


0 ure 8 uixjng 


OM^O^ 

rH 


LO 

rH 


oi 


rH 


II 0 IWB 0 S 


OON\fi 


r^ 


vO 


rH 


ssojdX^ 


ON\ON 
rH eg 


o 

to 


o 

rH 


rH 


0 IBMBI 0 Q 


OONO 

rH 


to 

rH 




rH 


^poqqnq 


O 00 o o\ 

rH 


LO 

rH 


00 


LO 


BUBquow 


rH 


rH 


to 


*3* 


III wodM 0 N 


rH LO tO tO 
rH 


to 


to 

rH 


to 


II JJOdM©^ 


O O tO VO 


vO 


to 


o 


I :uodM 0 N 


O O VO Cvj 
rH rH 


o 

rH 


vO 

rH 


to 


iibmbh 


o o r>* 


eg 

eg 


r^ 


o 


pUBiaJOd 


O LO 00 eg 


eg 


eg 


eg 



i 

i 




> tf, Z W 



§•8 JUS'S 

«s;s«s 

mm 



; 

i 

i 

) 




.Li 



s 



c n 



ft' 

0 O 0 

Jh 'H ^ 4-» ,Q 

8 §|S'C 



Mil’ 

^ e ? 

y O Q ^ 

p* S 

5 h <3 .0 



m 



r-1 w 

Si 



all 



t i & 

8J 2 



“ ■ ,§ 



Sn'oI 



&> 5 ^ 4 J 

2 0 §f eg 

3 _ 0 



42 



y b 



13 UrfJ ^4-J 



3 

8 8 >S 



0 



O 

r^i 



CM 



79 



appendix h 




I 



( 

* 

5 



i. 

t 

t 



t 



f 




i 



i 

i 



VERSION #2 IMPACT STUDY RESULTS 



80 

o 



vo n* o> o to ih 

©\° tO rH NJ tO 



to ^ to 

LO 



N Tt NO H VO 
rH tO 



to o eg to too 

Ng to rH M N4 



OON 
tO CVJ 



H O O O H 
tO iH 



i 



a> 



*tj o- 

33 w 

V) cd 
3 .8 
§ ,H 

a> 

se 

A cd 
•H 
</) M 

9 

15 a 

td w 

c.5 

HH V) 




I 

rH 

I 

Q) tj 

«s 

•H 

.S" 3 



fc 

£ 

Vi 



o- 
cd P, 

jg ,§ 

d> Vi 

I 



*d cd a> 

5 ?h pp 

0 ) q> +3 
Jh +-> 
•H44 
° 

O rH 
0> *H O 
> 4J E?) 
cd cd a> 
S 3 * 



to 



to 

o a> o 

52 ; >h a 



Cd .rO 



.'B 

i e w v» 
OH 2 3 O 

•S-8S8-S 



Jh 

d 



H.S* 



e g § i! a 

T> Ji *H • H 

ag 

•“■as "a 
sssa.a 

© Vi 4-> 
4-»m 0)fl Vi 
O 4-» © rt 
•HUP. 



8 
e* 

cd *H 
rP *d 



II ell 8*8 S 






s * J* 




cd 1( p . u *d . fl> 



vO 



o 



82 



A great deal 

Totally change my way of 
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Number of participants who described 
how they discussed and/or shared the 
workshop's materials and ideas with 
colleagues. 
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10. The Center is planning to con- 
struct intensive workshops in 
each of the areas listed below. 
In which of these would you be 
interested in participating? 
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FOLLOW-UP QUESTIONNAIRE 



EVALUATION WORKSHOP #1 
June 22-23, 1970 



To assist me in planning subsequent workshops utilizing the U.C.L.A. 

training materials, will you please respond to the following questionnaire 

Please base your responses on your experiences with, and reactions to, 

this week's evaluation workshop. 

1. Do you feel the workshop was beneficial to 
you as an administrator? 

2. Do you anticipate that the experience will 
have an effect on the way you plan in the 
future? 

3 . Was the concept of evaluation presented in 
the workshop consistent with your previously 
held ideas regarding evaluation? 

4. If not, have you now changed your concept of 
evaluation? 

( NOTE : There are two aspects of the work involved in the workshop: 

(a) the complexity of the tasks you performed; (b) the number of 
tasks you performed. Question 5 refers to (a) , complexity; question 
6 refers to (b) , amount.) 

5. Considering the level of the participants, how would you categorize 
the degree of complexity of the materials you used: 

TOO COMPLEX 5 

ADEQUATE II 

TOO SIMPLE J 

6. Considering the level of the participants, how would you categorize 
the amount of effort r equired: 

TOO MUCH 8 

ADEQUATE 8 

TOO LITHE 2 

of the following groups, 

check those for whom you think the materials might also 
be used. 

for those you do not check, please indicate whether it 
is because of the materials' complexity, simplicity, 
or inapplicability. 



7 . For each 

CD 



( 2 ) 



o 

ERIC 



16 


20 


YES 


NO 


17 


1 


YES 


NO 


13 


4 


YES 


NO 


4 


YES 


NO 



96 



a. 


Curriculum Directors 16 




• 




complexity 


simplicity 


not applicable 


1 


b. 


Other central office administrators 13 






complexity 1 


simplicity 


not applicable 3 




c. 


Project Directors 


(e.g., Title VIII, Interchange) 


15 




complexity 1 


simplicity 


not applicable 


1 


d. 


Building Principals 12 








complexity 1 


simplicity 


not applicable 


4 


e. 


Department Heads 


11 








complexity 1 


simplicity 


not applicable 


4 


f. 


Teachers 5 










complexity 1 


simplicity 


not applicable 


10 



Of the five phases of the evaluation process, were there any on which 
you feel either more time or less time could have been spent? 


a. 


Needs Assessment 


MORE 


11 


LESS 


b. 


Program Planning 


MORE 


13 


LESS 


c. 


Program implementation 


MORE 


12 


LESS 


d. 


Program improvement 


MORE 


11 


LESS 


e. 


Summative evaluation 


MORE 


12 


LESS 


Please rank the five sections in terms of: (a) their value to you as 

an administrator; and (b) their value to the district in terms of 
priority needs . 

VALUE TO YOU VALUE'- TO DISTRICT 


a. 


Needs assessment 




1 


1 


b. 


Program planning 




2 


2 


c. 


Program implementation 




4 


4 


d. 


Program improvement 




3 


3 


e. 


Sunmative evaluation 




5 


5 



10. In terms of time allocation for the workshop, rank the following in terms 
of your preference and the effectiveness of effort required. 

a. full day (8 1/2 hours) 5_ 

two half-days 4 



b. 



c. three half-days 1.5 



d. one full day and one half-day _3 

e. two full days 1.5 

Would you prefer to work in teams or as an individual? 
individual 2 



two man teams 7 



three man teams 8 



If follow up workshops were to be conducted, which of the following 
would you participate in and which would you want some or all of 



your staff to participate in? 










YOU 


YOUR STAFF 


a. 


developing behavioral objectives 
for instruction 


8 


11 


b. 


developing behavioral objectives 
for management 


15 


10 


c. 


writing proposals 


5 


12 


d. 


process of needs assessment 


12 


13 


e. 


personnel evaluation 


13 


9 
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or 









.'r 



• >V? ■ 









99 



